fix: define decoded AGC state and bound size arithmetic - #10249

What

Fixes undefined-behaviour and correctness defects in the AGC driver, guest libc, NGS2, IME, VideoOut, the shader recompiler and the relinker:

• Graphics::State, ShaderStages, ColorTarget and DepthTarget had members without initializers (scissor, cullMode, frontFace, blend, blendConstants, viewport, renderExtent, topology, hasColorTarget, negativeOneToOne, path, registerValue, the wave sizes, address, extent, format, bytes, componentMapping, clearDepth, clearStencil). Driver::sameDecode compares blendConstants and the sample masks, so a default-constructed State read indeterminate memory.
• Ngs2CustomModuleOption is a 4-byte size field; OptionWord read 4 bytes at offset 4, past the declared object. It now validates the guest-declared option size and returns the option error code when the option is truncated.
• Recorder indexed syncWaitedMs[source] (5 entries) and, inside CountThreadSync, counts[source] and waitedMs[source], before any range check.
• ThreadOwned lost the object when push_back threw while registering it.
• sceLibcMspaceCreate returned success when the map insertion failed.
• Size arithmetic was performed in 32 bits and widened afterwards in MultisampledDepthBytes, StorageTexture::sliceWindows, the vertex input table read, VideoOutDriver::processFlip and the Mach-O init stub displacement.
• Sequence::SubmitCompute copied the compute shader register and specials pointers with sizeof applied to pointer variables, which reads as a mistake. The copy now goes through std::uintptr_t, so the width is the pointer size by construction.
• fprintf received Registers::Reference proxies (Pm4 context trace) and uncast register values (draw capture dump).
• StencilReferenceUnused compared VkCompareOp against VK_COMPARE_OP_NEVER, which is 0 and therefore unreachable.
• FilterText computed type - 0x100 on std::uint32_t before checking type >= 0x100.
• ~Replay called its virtual Cancel/Wait unqualified, the tessellation attribute switch left InstanceInput/InstanceOutput implicit, and AdjustedDelta carried an unused cutStart parameter.

Tested

• OS: NixOS 26.05, x86-64, GCC 15.3, Release.
• cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release ..., cmake --build build --parallel 12 and cmake --build build --target libs --parallel 12 all succeed on top of current main; the patched .prx set is produced.
• ctest -R '^(mspace|agc_driver_pm4|agc_driver_draw_program_trace|guest_ime_filter_text|ngs2_core_render)$': 5/5 pass.
• clang-tidy -p build -checks='-*,bugprone-sizeof-expression' core/libs/prx/libSceJobManager/Export.cpp: reported Export.cpp:349 before the change, nothing after it.
• GPU tests were not run: the build environment has no Vulkan device, so the AGC and VideoOut tests with ANYPS5_REQUIRE_VULKAN=1, ANYPS5_NO_SHADER_CACHE=1 and the validation layer still need a run on hardware.
• python3 tools/check_conventions.py --base origin/main passes.
• python3 tools/pr_overlap.py --branch fix/agc-state-init-and-bounds reports file-level overlap with open PRs that implement unrelated features and fixes. The two hunks that duplicated open work were dropped: the DepthSurface tile-mode format is fix(agc): pass the tile mode of a mismatched depth plane copy as an int #7504, and the Sequence::Advance erase line is replaced by fix(job-manager): keep scheduling state in caller memory #10231/fix(job-manager): forward native profiler lifecycle events #10244.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Fixes: none
• Unblocks: none
• GPUs tested: n/a (GPU tests not run; no Vulkan device in the build environment)
• AI-assisted: yes